Introducing Feedback into a Mixture-of-Experts Model

The mixture-of-experts model is a static neural network architecture in that it learns input-output mappings where the output is directly influenced by the current input but not previous inputs. We explore a dynamic version of the mixture-of-experts model by introducing feedback into the architecture, enabling it to learn temporal behaviour. The model’s ability to decompose a task into static and temporal subtasks and to allocate those subtasks to relevant expert networks is examined. The performance of the model on learning the two subtasks involved in verbal counting is presented. A potentially useful outcome of the simulation is that simplifying the topology of the expert networks tends to improve allocation of subtasks to the most appropriate expert networks.